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ABSTRACT: Educational data science (EDS) is an emerging, interdisciplinary research domain that 
seeks to improve educational assessment, teaching, and student learning through data analytics. 
Teachers have been portrayed in the EDS literature as users of pre-constructed data dashboards 
in educational technologies, with little consideration given to them as active producers of data 
analytics. This article presents the case study results of an EDS program at a large university in 
Midwestern U.S.A. in which faculty and instructors were provided with access to institutional 
data and data analytics technologies in order to explore questions related to their classroom and 
departmental environments. Semi-structured interviews of program participants were conducted 
to examine the participants' experiences as practitioner researchers in EDS. The analysis showed 
that participants were motivated to participate to improve their learning and educational 
environments through data analytics, as opposed to developing a research agenda in EDS; that 
participants experienced a range of barriers related to data literacy; and that participant 
community support in addition to administrative support are vital to teacher-focused EDS 
programs. This study adds to a small but growing body of research in EDS that considers teachers 
as producers and not just consumers of data analytics. 

Keywords: Educational data science, learning analytics, academic analytics, data literacy, 
practitioner research 


1 INTRODUCTION 

While there has never been a dearth of student data collected by universities, the last decade has seen 
an increase in the types of data captured as well as developments in analytics capabilities, which have 
enabled new avenues for data-driven inquiry into the classroom environment and student learning 
(Slade & Prinsloo, 2013). In this data-driven environment, educational data science (EDS) has emerged 
as a multi-disciplinary community of researchers (Piety, Hickey, & Bishop, 2014). Due to the increased 
ubiquity of educational technologies, such as learning management systems (LMSs) and MOOCs, 
educational data scientists have been able to move beyond institutional assessment and into classroom 
assessment (Reyes, 2015). The communities that have emerged within EDS include academic analytics 
(Goldstein & Katz, 2005), educational data mining (Baker & Yacef, 2009), and learning analytics 
(Siemens, 2012). 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


193 



JOURNAL OF LEARNING ANALYTICS 


S °'iLAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 

(2016). Teachers as producers of data analytics: A case study of a teacher-focused educational data science program. Journal of Learning 
Analytics, 3(3), 193-214. http://dx.doi.org/10.18608/jla.2016.33.10 

The research in EDS has primarily focused on the application of data analytics to improve student 
learning, academic performance, and institutional practices. In this research, the teacher has been 
predominately considered as a user of technologies that display analytics about the learning 
environment with little consideration given to the teacher as a possible producer of analytics. Research 
supporting this view has addressed how teachers use and integrate the data analytics from dashboards 
and other similar technologies into their student and classroom assessment and subsequent 
interventions (Verbert et al., 2014). In the current educational climate that pushes for increased 
institutional assessment, EDS is in the position to empower teachers not only to use educational 
analytics tools, but to become active participants in EDS. While the literature on teachers as participants 
in EDS is starting to gain traction in the learning analytics community (Clow, 2014; Ferguson et al., 2014), 
to date there have been few empirical studies focusing on teachers' motivations to participate in EDS 
and on the barriers they face while engaging in data analytics. 

In this paper, we address these gaps in the literature through the results of a study of university 
instructors and faculty who participated in a teacher-focused EDS program that positioned teachers not 
as users of analytics, but as active participants in the production of data analytics. Based on interviews 
with the program's participants, we found that regardless of the participants' previous data analytics 
backgrounds, all had experienced some type of barrier related to data literacy. Data literacy of teachers 
has been addressed in the learning analytics literature in relation to the ability to interpret and use 
analytics for assessment (Pea, 2014); however, our findings suggest that data literacy in EDS is an 
intricate web of interconnected skills necessary for identifying, organizing, analyzing, and summarizing 
data. In addressing these barriers in the program, participants valued administrative support, which was 
integral to the completion of the participants' projects. However, the participants stated that they 
wished that more emphasis had been placed on inter-project collaboration. Furthermore, the results 
suggest that these types of programs should not limit teacher inquiry to a particular EDS community, as 
the participants were motivated to use data to learn more about their educational environment than 
they were to identify with or develop a research agenda in a particular EDS community. Based on the 
study's results, we present the best practices for teacher-focused EDS programs and argue for emphasis 
on supporting teacher's inquiry interests as they relate to the different EDS communities; data literacy 
training that addresses not only data interpretation, but data access and analysis; and developing a 
collaborative participant community. 

2 RELATED WORK 

2.1 EDS Communities 

Piety et al. (2014) have identified four emerging communities in EDS, which they state are 
academic/institutional analytics, learning analytics/educational data mining, learner 
analytics/personalization, and systematic instructional improvement. The academic analytics and 
systematic instructional improvement communities are similar to the data use practices associated with 
institutional research, in that they focus on macro-level analyses of the units within higher education, 
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such as colleges, schools, or departments (van Barneveld, Arnold, & Campbell, 2012). Furthermore, 
academics analytics focuses on organizational concerns, such as how students progress through the 
educational system regarding retention and persistence in a major or program (Piety et al., 2014). 
Learning analytics and educational data mining, on the other hand, take a micro-level approach to 
understanding the learning process through the "the measurement, collection, analysis, and reporting of 
data about learners and their contexts, for the purpose of understanding and optimizing learning and 
the environments in which it occurs" (Siemens & Long, 2011, p. 34). Learner analytics, while still focusing 
on the classroom environment, differs from learning analytics in that rather than focusing on the 
learning process, researchers take a macro-level view of the classroom to understand the differences 
among students' cognitive traits in relation to academic success (Piety et al., 2014). 

The communities of learning analytics and educational data mining in EDS have quickly developed into 
academic disciplines with their own journals and conferences (Siemens, 2013); however, in practice the 
systematic implementation of EDS outside of institutional research offices and educational researchers 
has faced challenges. Picciano (2012) notes that higher education lacks qualified professionals, outside 
of educational data scientists and institutional researchers, trained to use and understand large sets of 
student data. Regarding learning analytics, Ferguson et al. (2014) address this deficit of analytics 
professionals by suggesting that universities should begin to view the development of human resources 
as imperative for organization-wide learning analytics adoption. In practice, however, Bichsel (2012) 
suggests that while university administrators support analytics initiatives, the costs of such initiatives 
have been prohibitive. Furthermore, Bichsel (2012) concludes that "many institutions view analytics as 
an expensive endeavor, rather than as an investment" (p. 3). The costs associated with analytics is not in 
relation to the technologies associated with analytics, rather the costs related to the recruitment and 
development of data analytics professionals (Goldstein & Katz, 2005). 

2.2 Data Literacy and EDS 

As universities and colleges begin to address these data analytics barriers related to human resources 
and analytics adoption in higher education, they must focus their attention on ensuring that institutional 
staff possess and develop the skillsets required for data analytics (Prinsloo, Archer, Barnes, Chetty, & 
van Zyl, 2015). In addition to this institutional perspective, Selwyn (2015) argues that educational 
researchers too are facing challenges regarding the development of the necessary skills to "engage 
effectively and insightfully not only with the social uses of data, but the data themselves" (p. 14). While 
the data analytics capabilities of these two groups, institutional staff and educational researchers, is 
certainly important to address, teachers must begin to be considered as key actors in EDS, particularly in 
the assessment of teaching and student learning (Crisp, 2012). With the development of new 
assessment techniques, particularly e-assessment, teachers are hindered by the lack of training 
opportunities, which results in low levels of teachers' computer and technical literacy required to 
engage effectively with these techniques (Guardia, Crisp, & Alsina, 2016). 
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Pea (2014) argues that attention must be paid to the barriers that teachers face when presented with 
analytics to make data-driven decisions about the classroom environment. This is important to consider 
since EDS is not only a concern of analytics professionals and academic researchers, but also teachers 
who are positioned to blend data-driven decision making with human judgement to impact the learning 
environment (Siemens & Baker, 2012). As a result, universities and colleges not only need to focus on 
the development of analytics professionals, they must also begin to address the teacher's role in the 
data analytics process given that learning analytics and learner analytics depend "upon the ability of a 
teacher to quickly make sense of data visualizations presented in learning dashboards" (Pea, 2014, p. 
40). 

Common throughout the research in the EDS communities is the role of the teacher in the analytics 
process. Teachers are portrayed as users of analytics technologies, such as data dashboards. Concerning 
these data dashboards, teachers are provided with visualizations of aggregated data about their 
students and their performance in the classroom environment (Verbert et al., 2014). As users of 
analytics technologies, however, teachers still must possess the necessary data literacy skills to use and 
understand data to inform their decision making (Mandinach & Gummer, 2013). The skills associated 
with data literacy include those necessary to identify, collect, organize, analyze, and summarize data 
(Mandinach, Honey, Light, & Brunner, 2008). The research on teachers in EDS to date has primarily 
focused on the latter two of these skills, meaning that this research has explored how teachers use and 
interact with the analytics provided to them and how they make use of these analytics to inform their 
classroom teaching and student assessment. Examples of such studies include teachers' adoption beliefs 
of analytics technologies (Ali, Asadi, Gasevic, Jovanovic, & Hatala, 2013), their use practices of real-time 
evaluation tools of student learning (Chounta & Avouris, 2014), and frameworks for developing learning 
analytics systems for teachers (Dyckhoff, Zielke, Bultmann, Chatti, & Schroeder, 2012). The research is 
lacking on the data literacy skills necessary for teachers using analytics in the classroom environment. In 
this work, we address the issues related to data literacy by examining the types of barriers that teachers 
face when given the student data and tools required to conduct data analytics. 

2.3 Teacher-Focused EDS 

While EDS research has primarily addressed teachers as users of educational data analytics, a number of 
programs have developed in EDS that have positioned teachers not only as users, but also as producers 
(Ferguson et al., 2014; SLAM, 2014). The University of Michigan's Student Learning and Analytics at 
Michigan (SLAM) provided faculty in the program with access to student data and training through 
seminars to introduce them to learning analytics tools and student data (NMC, 2013). The program 
included not only faculty, but graduate students as well, and had the support of a larger institutional 
task force, the Learning Analytics Task Force (SLAM, 2014). In another program, the Data Wranglers at 
the Open University, academic staff worked in collaboration with the university's seven academic 
faculties to participate in a learning analytics initiative (Clow, 2014). Through this program the Data 
Wranglers conducted analytics using a variety of data sources, and they served as data liaisons to the 
academic departments by providing them with actionable reports. Furthermore, the Data Wranglers 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


196 


JOURNAL OF LEARNING ANALYTICS 


S °'iLAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 

(2016). Teachers as producers of data analytics: A case study of a teacher-focused educational data science program. Journal of Learning 
Analytics, 3(3), 193-214. http://dx.doi.org/10.18608/jla.2016.33.10 

were encouraged to foster an interest in learning analytics and to develop a learning analytics 
community of practice in the departments (Clow, 2014; Ferguson et al., 2014). 

In this study, we examine a teacher-focused EDS program similar to those outlined above (Clow, 2014; 
Ferguson et al., 2014; SLAM, 2014) to understand the experiences of teachers actively participating in 
EDS. We developed three research questions to guide this qualitative study: 

RQ1: What factors motivate faculty to participate voluntarily in teacher-focused EDS 
programs? 

RQ2: What types of data access, use, and interpretation barriers and challenges do 
faculty encounter in teacher-focused EDS programs? 

RQ3: In what ways could future teacher-focused EDS programs provide support to aid 
participants in the educational data analytics process? 

3 CASE STUDY BACKGROUND 

At a large Midwestern university in the U.S.A., a teacher-focused EDS (TEDS) program of staff and faculty 
was created to enable the exploration of student engagement, retention, and success through the use 
and analyses of student data. The program had the support from the university's centre for teaching and 
learning, the research and assessment office, and the Office of the Vice Provost for Undergraduate 
Education, and was a part of a larger campus-wide initiative to further student success through 
educational data science. Given that this was the TEDS program's first year in existence, according to the 
program director the administration recognized that there would be issues related to the faculty's 
production of data analytics and the administration's data support. As such, this first year of TEDS was to 
serve as a pilot for future iterations of the program. 

According to the TEDS program director, the purpose of the program was to provide an outlet for faculty 
to understand and use institutional data to construct data analytics projects that focused on answering 
questions related to student success. The program was rooted in the scholarship of teaching and 
learning, and more specifically was based on the concept of faculty learning communities. A faculty 
learning community is comprised of cross-disciplinary faculty that participate in a program dedicated to 
improving teaching and student learning (Cox, 2004). The TEDS program solicited faculty from a number 
of disciplines who held a teaching role at the university, which included tenured and tenure-track 
faculty, teaching faculty, and university staff. Graduate students were allowed to participate so long as 
they were part of a faculty-led project. The projects chosen for TEDS were related to improving teaching 
and learning through the application of educational data analytics. 

Beginning in December 2014, the one-year program provided participants with access to institutional 
student data, $2,000 in research funds, and a license for Tableau, a data visualization software. The 
participants in the program were required to attend regular meetings with a program director from the 
university's centre for teaching and learning. The meetings provided an outlet for participants to share 
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the status of their ongoing research, connect with other participants, interact with university staff who 
routinely conduct analytics with student data, and learn about the current state of educational data 
science. The nine projects accepted for the 2014-2015 TEDS program included those that can be defined 
as learning analytics/educational data mining, learner analytics, and academic analytics (Piety et a I., 
2014). 

4 METHODS 

4.1 Participant Selection 

The study took place in the spring of 2016 and focused on participants who took part in the 2014-2015 
program, which ended in December 2015. Prior to conducting the study, the first author attended three 
of the TEDS participant meetings in the fall of 2015. In addition, the first author met with the program 
director on two occasions, and had informal conversations with the program's participants. To solicit 
participants for this study, the faculty, staff, and graduate students listed in project completion reports 
found on the program's website were invited to participate in semi-structured interviews. There were a 
total of 24 participants listed as having participated in one of the nine projects. A description for all of 
the projects can be found in Table 1. The analytics type category in the table refers to the EDS 
community that the project most aligns with. 

Upon speaking with the participants, we further broke down our sample into 17 core participants and 7 
peripheral participants. We defined core participants as those whose involvement in the program 
started from the beginning and continued to the end, and who were actively involved in their project. 
The participants considered peripheral included those brought in near the end of projects to aid in data 
analysis or in writing up the TEDS completion report. In our interviews, we focused on the experiences 
of 10 core participants because they had a comprehensive understanding of the TEDS program, and 1 
peripheral participant who was able to speak to the overall experiences of their project's core 
participants. These core participants were chosen because they played significant roles in their 
respective TEDS projects. In total, we ended up interviewing 11 participants that represented 7 out of 
the 9 total projects. The participants represented tenured and tenure-track faculty and non-tenure track 
lecturers (4), university staff with teaching responsibilities (4), and doctoral students (3). When reporting 
the results, we use PXX to denote the participant number and PRX to denote project number. 

4.2 Data Collection and Analysis 

We conducted semi-structured interviews with the participants that ranged from 30 minutes to just over 
1 hour. Eight of the interviews were conducted one-on-one between the first author and the participant. 
The interviews took place from January to March in the spring of 2016. One of the interviews was 
conducted in a focus group, which included all of the participants representing one project. All of the 
interviews were audio-recorded and transcribed for data analysis. The interview questions focused on 
the following areas: teacher evaluation practices; motivations to participate in TEDS; barriers and 
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limitations faced during the program; perceptions of using analytics as an evaluative tool; and program 
expectations and results. Appendix A shows the semi-structured interview protocol. 

Table 1. TEDS Project Descriptions 

Project Community Project Description Data Used Methods 


PR1 Academic Analytics Retention and Institutional and Descriptive statistics 

graduation rates surveys 


PR2 


Academic Analytics 


Major transition by Institutional, 

major types, surveys, 

demographics, and interviews 

academic performance, 
and high school test 
scores 


Descriptive statistics 


PR3 


Learning Analytics Course interaction and LMS data and Network analysis 
course engagement institutional 
levels 


PR4 


Academic Analytics 


Major selection by Institutional 
academic performance 


Descriptive statistics 


PR5 


Academic Analytics Influences on academic Institutional and Descriptive statistics 
performance, and departmental 
academic performance 
on graduates' salaries 


PR6 Academic Analytics Major selection and Institutional 

graduation rates by 
academic performance 


Descriptive statistics, chi- 
square, and binary logic 
regression 


PR7 


Learning Analytics 


Academic performance 
and course interactions 


LMS data and Descriptive statistics, 

institutional temporal analysis, 

geospatial, and topical 
analysis 


PR8 


Academic Analytics 


Students' levels and Institutional 
academic performance 


Descriptive statistics and 
hierarchical linear 

modelling 


PR9 Academic Analytics Major selection by Institutional Descriptive statistics 

academic performance 
and demographics 


The interview responses were analyzed with an open coding approach through an iterative process 
(Corbin & Strauss, 2014). The coding process was not constrained by the structure of the interview 
questions but was rather completely open-ended. The first author initially openly coded all the data 
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collected, which generated 28 codes. Example codes included skills deficiencies, data barriers, and 
institutional support. The codes were then discussed between the two authors to iteratively merge, 
refine, and identify the most prominent themes. Five rounds of iterations eventually converged into 
three themes as presented in the results section: 1) motivations to participate; 2) data access, use, and 
interpretation; and 3) administrative and participant community support. 

5 RESULTS 

5.1 Motivations to Participate in TEDS 

In the TEDS program, the production and application of data analytics lies in the hands of willing 
instructors to adopt analytics as an evaluative tool of student learning, educational outcomes, and 
department performance. We found that participants were primarily motivated to use student data to 
answer questions related to improving their educational environment. However, prior to TEDS, the 
participants primarily relied on traditional assessment tools to evaluate student learning and course 
performance. While four of the participants (PI, P2, P9, Pll), had some prior experience using 
institutional data for student and course assessment, the remaining participants had not. 

I didn't know a lot of that existed until the Fellowship. I didn't realize we had the ability to access 
it. Not so much that I didn't know it existed, I realize that it's collected, but I hadn't really thought 
about the application to my work. (P6) 

These traditional assessment tools include evaluating student academic performance through grades 
associated with assignments, exams, and quizzes, and final course grades; and institutional course 
evaluations distributed at the end of each semester. The participants also expressed that they had relied 
on anecdotal evidence provided through conversations with students and their classroom experiences 
(PI, P6, P7, P9). In P9's study, they examined the performance of students in upper-level law courses in 
P9's department and the students' performance in lower-level law courses. Prior to TEDS, P9 had spoken 
with students who had stated that taking certain courses had helped them be successful in P9's course. 

While these three types of assessment — academic performance, course evaluations, and anecdotal 
evidence — have provided the participants with the substantial information to inform their teaching, the 
TEDS participants expressed that what had propelled them to participate in the TEDS program was the 
prospect of improving their educational environment and teaching through a systematic study of their 
students, the classroom, and the broader learning environment. Out of the participants we interviewed, 
nine of them (PI, P2, P3, P5, P6, P7, P8, P9, Pll) explicitly stated that they participated in the program 
to answer questions related to the improvement of their educational environment. 

Our goal is to educate, instruct, and teach our students. So when the learning analytics program 
came up, it was kind of just a no brainer. Let's see what we can do, more than just anecdotal 
evidence and more than just these course evaluations; let's take a big look at the whole program 
and this program allowed us to do that. (P7) 
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Six of the projects (PR1, PR2, PR5, PR6, PR8, PR9) focused on a macro-level view of improving teaching 
and learning as they were motivated to investigate the overall effectiveness of individual courses or 
their department curriculum. The research questions in these projects were related to future student 
academic performance, retention, on-time graduation, and curriculum development. For example, in 
P7's project their group was motivated to participate because they wanted to evaluate the effectiveness 
of the introductory courses in their department and to validate their view that providing these courses 
with adequate funding and qualified instructors is imperative to student success. 

I think sometimes we get in this, "oh I teach this major" but no, we are all part of a bigger 
department, and success in the department is success for everyone. We can't let the curriculum 
drop because then if we change the curriculum, the curriculum fails and that's going to lead to 
further decrease in enrollment in our department. (P7) 

5.2 Data Access, Use, and Interpretation 

While the participants in TEDS were primarily motivated to improve student learning and teaching, most 
of the participants were entering new assessment territory. The results found that only one participant 
(Pll) had used institutional data to conduct a multi-semester analysis. As such, participants ran into 
barriers regarding the access, use, and interpretation of the institutional student data. Based on the 
interviews, we identified two areas where the participants experienced data barriers: data access and 
data analysis. 

5.2.1 Data Access 

Participants were given access to one of two types of data: those stored in the university data 
warehouse, and those from the university's learning management system. For the projects that used 
student data stored in the university data warehouse (PR1, PR2, PR5, PR6, PR8, PR9), prior to requesting 
the data the participants attended an hour-long meeting with the office supplying the data to learn 
about what student data could be retrieved and how to request the data. However, the participants 
reported that even following this overview session, they were still unclear about what was available to 
them and how to go about requesting and accessing the data. In addition, the participants felt that the 
data sets were much more complicated than they had expected as the data sets included many 
institution-specific data that could not be understood intuitively. 

There are lots of indicators in data fields, titles of data fields that are not at all self-evident. I am 
speaking as an outsider, so I am guessing that other outsiders might feel the same. (P2) 

The participants expressed issues in communicating data needs (PI, P2, P6, P7, P8, P9, P10, Pll). For PI, 
the data access process resulted in multiple communication breakdowns with the participant having to 
routinely go back to the data providers to clarify their project's data needs. For another participant 
(P10), one of the few who had previous experience in conducting analyses on large data sets, the 
process of accessing the data and communicating their needs took time, but the process was not as 
laborious as for those with no prior experience. 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


201 


JOURNAL OF LEARNING ANALYTICS 


S °'iLAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 

(2016). Teachers as producers of data analytics: A case study of a teacher-focused educational data science program. Journal of Learning 
Analytics, 3(3), 193-214. http://dx.doi.org/10.18608/jla.2016.33.10 

Those guys were stretching as far as they could to understand my stuff and my project and what I 
was asking for. They are amazing, I don't want what I am saying to come off as a criticism of 
them... We speak two different languages, I was straining as far as I possibly could to understand 
what they were asking me. (PI) 

It took several weeks of communication. They sent us different data sets and we used them and 
said "these work and that doesn't." It wasn't a barrier. Collaboration takes that kind of 
communication and alignment of expectations. (P10) 

Even still, navigating the data access process proved to be complicated, and forestalled many of the 
projects' analyses of the data. While most of the participants requested anonymized data, P6's project 
required student identifiers since their analysis sought to correlate student data to a departmental data 
set on students' job outcomes. Although the initial data set received by the institutional reporting office 
included student identifiers, these had been anonymized, thus making the data set unusable for the 
participant. One participant (Pll), who joined their project towards the end of the program, stated that 
when they were brought on, the other project participants had not had time to analyze their data in the 
intended way prior to the program ending, as the process of clarifying their data needs and then 
accessing the data took longer than expected. 

I didn't realize that they had modified those. I found that out a little late in the process, mostly 
because I had assumed I had done something wrong. I kept looking for how I could fix it, then I 
realized we just got our wires crossed about how I was going to use that. They were super 
receptive to fix it once we knew what the issue was, it just took me a little longer to get there. 

(P6) 

They got it in batches and then there were some issues and delays in getting access to the data. 

They didn't have time to look at it and cut it in different ways to try to analyze it. (Pll) 

For the participants who sought access to data from the university's learning management system 
(LMS), issues of data access inequality arose. While the office supplying the institutional student data 
worked in conjunction with the TEDS program, access to the LMS data was not affiliated with the 
program. Thus, program participants had to try to access the data on their own. In total, three projects 
sought to use LMS data (PR3, PR5, PR7), yet only two of the projects were able to access the LMS data in 
the end. For the projects able to access the LMS data (PR3, PR7), access was not equal, since one group 
had direct access to the LMS database, while the other had mediated data access. 

It's not necessarily equal access. I know one of the groups worked with LMS data that they had as 
well from the same Red Shift Database, but they weren't given direct access. (P8) 

5.2.2 Data Use and Interpretation 

The participants began their projects with various levels of data analytics skills, with some having had a 
great deal of background in data analysis (P6, P8, P10). However, for others their educational 
backgrounds precluded their ability to conduct the data analytics necessary to answer their research 
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questions presented in their initial TEDS grant proposal. According to the program director, the TEDS 
administration was aware that participants might not necessarily have the requisite data analytics skills. 
While the TEDS program provided limited training on accessing the data, formal training in data analytics 
was not provided. As a result, the program participants had to make do with the skills they had coming 
into the program, learn the skills on their own during the program, or hire graduate students to conduct 
the analysis. 

It was completely new to all of us, in fact that was maybe the only frustration we had with the 
whole thing is figuring out the steps. How do you get the data? Then when we get the data, 
there's a hundred and fifty variables, there's three thousand entries. How do we analyze this? 

(P7) 

Of the participants interviewed, nine (PI, P2, P3, P4, P6, P7, P8, P9, Pll) explicitly stated that they or the 
other participants in their project did not have the statistical or analytical skills to conduct the analysis 
that they had initially intended at the start of the program. The skills that the participants stated that 
they lacked included multivariate statistical analysis (P6), textual and discourses analysis (P3, P4, P5), 
predictive modelling (P8), and data visualization (P5). The lack of analytics and statistical skills resulted in 
the inability to answer some of the projects' proposed research questions (PR1, PR3, PR5). 

I thought I had a lot of skills coming in that I thought were going to apply more here than they 
did. I learned a lot about where my gaps were and how try to overcome them. (P6) 

For the projects that could afford it (PR3, PR6, PR8, PR9), either through the money provided by the 
TEDS grant or through other research funding, graduate students were hired to assist with the analysis 
of the data. Other projects, however, had to rely on their previous statistical and analytical knowledge 
(PR1, PR5, PR7). When asked how the participants addressed their lack of statistical skills in their 
project, P2 stated that their group applied descriptive statistics and were cognizant not to overstate 
their findings. 

Mostly through hedging and hopefully, not in a dishonest way, but not overstating what we think 
we can state. Like, this seems to indicate a pattern, or interestingly this percentage of students 
graduated in five years. (P2) 

The issues with statistical analyses are similar to those the participants encountered using the data 
analytics software, Tableau (PI, P2, P4, P6, P7). As a part of the program, the projects were given one 
professional license to the Tableau Desktop to aid in their analyses and visualization of the data. The 
software had not previously been used by any of the participants; therefore, the TEDS program provided 
the participants with a tutorial of the software during one of the program's meetings. However, not all 
of the participants were able to attend the training, nor was it a hands-on training. Each project, aside 
from two (PR7, PR8), which chose instead to use technologies they were familiar with, was given access 
to the software. While the participants attempted to use Tableau, in the end all of the participants who 
had access to the professional license ended up using a data analysis software that they were more 
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familiar with either entirely or in addition to Tableau (PI, P4, P6, P7, P9, P10), examples include Excel 
(PI, P2, P6), Gephi (P4), or R (P10). 

Participants were asked how the barriers they faced regarding learning analytics could have been more 
systematically addressed by TEDS. They stated that they would have valued more formal training as a 
part of the TEDS program in lieu of the brief overviews on data access and Tableau, and the informal 
one-on-one trainings with the program's data providers (PI, P2, P6, P7, P8, P10). Given that all of the 
projects faced some struggle in accessing and interpreting the data, the participants stated that training 
in data access, data analysis, and data visualization would have been helpful in their projects. 

Going forward, having some upfront familiarity with the kinds of data that can be collected, 
through what means, tools for analyzing, tools for sharing, basically a kind of overview of the 
whole process you might imagine would be helpful. (P2) 

Maybe even having cohorts of people that are learning the same skills, like they are either going 
through courses or trainings together so that they can bounce off of with everybody so that they 
can collaborate more. (P8) 

5.3 Administrative and Participant Community Support 

Developing teacher-driven educational data analytics through collaboration and support was one of the 
primary goals of TEDS. The support that the participants identified came from the participants' project 
collaborators, participants from other projects, graduate students, university administrators, and 
university staff. This support structure embedded in the program was attractive to the participants who 
were new to data analytics. 

I am a lecturer so I am not a research faculty, so for me it was important to have the support 
knowing who I go to to get data, what rules I have to follow to adhere to research protocols. (P6) 

The interviews revealed two primary types of support that aided in the participants' projects: 
administrative support from staff and administrators in charge of TEDS; and support from among TEDS 
participants. Although these two types of support contributed to the success of the program, a number 
of those interviewed (P2, P3, P5, P6, P8) specifically addressed ways that the TEDS participant 
community could have been improved. 

5.3.1 Administrative Support 

Of the participants interviewed, each commented to some degree that they valued the administrative 
support provided during the program, and that the support contributed to the completion of their 
projects and the success of the program overall (PI, P2, P3, P4, P5, P6, P7, P8, P9, P10, PI 1). The 
interviews revealed that the participants received two types of administrative support: from university 
administrators funding and managing the program; and from university assessment office staff who 
provided access to the institutional student data. These two administrative groups communicated 
frequently with and provided consistent support to the participants. 
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At another university I've been a part of a somewhat similar group and I would say it was much 
more high pressure and it was very much a kind of a mini grilling session and it just wasn't that 
productive... It was a very different feel to it. (P5) 

Although the participants had various data issues in accessing, using, and interpreting the data, of the 
projects that used student data from the university's data warehouse, they received consistent support 
from the office in charge of access to that data. The participants stated that when they routinely went 
back to that office for clarification in understanding their data sets, their questions and concerns were 
always addressed. This support throughout the data access step in the TEDS projects proved vital to the 
completion of many projects. Furthermore, the office supplying the data offered one-on-one assistance 
to groups having issues requesting data or making sense of data using the data analysis tool, Tableau. 

Again, those guys from [the data providers] were so great. They walked us through how to use 
[Tableau]. They even came over here one day and showed me on my computer. We would go to 
visit them and I'd be like "but, when I pull it up on mine it doesn't look quite like yours." So they 
came here and showed me how to interact with it. (PI) 

5.3.2 Support from TEDS Participants 

The TEDS participant community was comprised of faculty, staff, and graduate students from various 
academic disciplines and methodological backgrounds. In spite of these differences, the participants 
noted that the TEDS program meetings provided an avenue through which the participants were able to 
share their experiences throughout their projects (PI, P5, P6, P7, P8). The participant community served 
as a motivator for participants not experienced in data analytics (PI); it enabled participants to hear 
about the data analysis techniques employed by other TEDS projects (P5, P6, P7); and exposed them to 
others with similar research interests (P8). 

Having a community, a supportive community, being part of a project where other people were 
also doing their projects. That was really extremely motivating to me. (PI) 

I learned a lot being a part of that Fellowship and going to those regular meetings. Not even 
asking questions, but just hearing what questions were asked kind of set off light bulbs for me of 
"oh I can do this, I can't do that." And different ways of thinking about looking at the data. (P6) 

As important as participant support was during the meetings, the participants expressed that being a 
part of a participant community did not extend past the meetings. Rather, after the meetings adjourned 
participant communication was contained within each of the participants' projects. This lack of 
continued participant community support proved problematic for two of the participants (P6, P9) as 
they had applied to TEDS as individuals and not as a part of a group. For P6, this reduced their ability to 
share ideas as they worked through using student data and data analytics technologies, such as Tableau, 
for the first time. For the other solo project, the participant (P9) struggled through the project until 
compelled to seek out a graduate student to assist. 
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While the meetings provided a setting for participants to recount their experiences in the program, 
inter-project collaboration, though not necessarily discouraged, was not built into the program. This 
feeling that the community existed only during the meetings and not beyond, prompted some 
participants to suggest that for TEDS to continue successfully beyond its current iteration, the program's 
stakeholders need to emphasize supporting and bolstering the community amongst the program's 
participants (P2, P3, P5, P6, P8). This type of community-based approach would have enable participants 
to know more about the data analytics abilities of participants from other projects for the purpose of 
possible future collaboration. 

I know some who are in all these other departments and they have these general sets of analysis 
techniques, but it is hard to say I want to collaborate with you just because, without reading all 
of their projects in depth and knowing the techniques that they applied, but there is no real way 
for us to communicate easily what skills we have to bring to the table. (P8) 

6 BEST PRACTICES 

We have outlined TEDS participants' motivation to participate, their data access and analytical 
challenges, and the support they received in the program from the TEDS administration and the 
participant community. Below we situate our findings within current understandings of EDS and further 
offer recommendations of best practices to help generate a sustainable analytics program that focuses 
on empowering educators to conduct their student data analytics. 

6.1 Participant Motivation 

6.1.1 Confirmation of Literature 

Based on our results, we found that the TEDS participants were primarily motivated to improve the 
learning environment through the generation and use of student data analytics. This finding aligns with 
other research on teachers' perspectives on the use of data analytics to understand the classroom 
environment. In a survey of educators' perceptions of learning analytics, Greller and Drachsler (2012) 
found that the educators felt that the primary objectives of learning analytics are to improve the 
students' learning experience and the learning environment. They stated that learning analytics should 
be about providing personalized learning experiences for learners, supplying timely information about 
the learners' progress, and providing educators with detailed information about the learner's context. 
Our results suggest that the views of the participants in TEDS align with researchers in the learning 
analytics community who have been vocal in emphasizing that the value of learning analytics lies in 
improving the learning environment and not solely in analytical and technical capabilities (Dringus, 
2012; Gasevic, Dawson, & Siemens, 2015). 

6.1.2 New Insights 

The results from the study suggest that the participants in TEDS were more interested in conducting 
analytics specific to their learning environment and broader academic contexts than participating in the 
emerging EDS communities. Furthermore, although some of projects aligned with the research 
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conducted in learning analytics, the teachers were also motivated by types of inquiry more similar to 
other communities associated with EDS. While a few of the projects conducted analytics based on 
student learning at the classroom-level (PR3, PR7), the remaining projects were more similar in scope to 
the research associated with academic analytics. Future teacher-based EDS programs should encourage 
teachers to explore various inquiry types in relation to their educational environment. 

We understand that one of the limitations of our study is that the TEDS participants actively chose to 
participate in the program. Yet, the purpose of EDS programs like TEDS are to provide interested 
educators with the opportunity to use student data to understand the educational environment in 
which they are embedded. TEDS is not just a program that provides educators with the resources for 
conducting data analytics, it is also a program that empowers educators interested in learning more 
about their students' learning, their students' academic performance on major retention, their own 
teaching, and other questions related to their departments, by enabling them to explore these issues 
outside of the traditional means of assessment. This suggests that TEDS's focus on providing educators 
with student data to conduct their own analyses can enable future EDS programs similar to TEDS to 
conceptualize the educators as active producers of analytics as opposed to viewing them as users of pre¬ 
constructed data analytics embedded in educational technologies. 

6.2 Data Literacy and the Teacher 

6 . 2 .1 Confirmation of Literature 

In EDS literature, the teacher-as-user in the learning analytics is portrayed as an interpreter of 
meaningful information from the analytics about student performance to identify possible intervention 
(Greller, Ebner, & Schon, 2014) through the use of dashboards that aggregate data and display student 
analytics (Chounta & Avouris, 2014; Dyckhoff et al., 2012; Verbert et al., 2014). In TEDs, the participants 
are not just users of data analytics, but are more broadly conceptualized as actors who produce and 
then use analytics. The results of this study suggest that the participants encountered two types of data 
issues: data access barriers and data analytics skills deficiencies. These issues are not unique to TEDS nor 
to learning analytics or academic analytics. The fact that in higher education there has been a deficit of 
individuals capable of producing analytics to improve teaching and learning has been noted for some 
time (Goldstein & Katz, 2005; Ferguson et al., 2014; Pea, 2014; Picciano, 2012). If universities and 
colleges are facing issues in finding analytics professionals and implementing learning analytics system- 
wide, it is no surprise that the participants of TEDS encountered issues when attempting to conduct 
their analyses. 

We agree with Pea (2014) that data literacy for educators is an important step in the evolution of 
learning analytics and EDS. Our results further demonstrate that data literacy is about more than just 
one's ability to read and interpret data. Data literacy required in TEDs is not just similar to other types of 
literacy, including information and statistical literacies, it actually encompasses the skills related to these 
other literacy types. The skills that encompass this broader definition of data literacy include those 
necessary to be able to identify, collect, organize, analyze, and summarize data (Mandinach & Gunner, 
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2013; Mandinach et al., 2008). By viewing data literacy in this light, one finds that further skill sets are 
required for teachers to become data literate and to engage more fully in teacher-focused EDS 
programs. 

The results from the interviews suggest that the TEDS participants underestimated the importance of 
data literacy at the start of program, in particular the breadth of technical, analytical, and statistical skills 
necessary to conduct their analyses. From the results, we found that the participants' expectations 
concerning the questions they proposed to answer during TEDS did not fully match up to what they 
actually ended up answering in their final analyses. As one participants put it, "I certainly had ambitions, 
but my ambitions were greater than our accomplishments" (P3). The participants either undervalued or 
were unaware of the various skills necessary for data literacy, and though the participants possessed 
some skills associated with data literacy, they were unknowingly deficient in one skill or a combination 
of them related to data access, analysis, and visualization. 

6.2.2 New Insights 

Data literacy could prove to be one of the more challenging barriers to teacher-focused EDS programs, 
since for teachers to engage in data analytics effectively, data literacy should be viewed as an integral 
part of the process in transforming teachers into analytics producers as opposed to analytics users. Data 
literacy of teachers should be addressed by providing training opportunities in the skills necessary to 
effectively engage in educational data science. Data literacy training should include an introduction on 
student data that focuses on how to request and interpret the data sets. The latter is of particular 
importance given the various types of data structures that constitute a university's institutional data, 
such as those determined by individual schools and those set by national standards. As the results 
suggest, the projects that used student data from the university data warehouse did not have issues 
accessing the data in the end, but rather had trouble assessing the precise data that they needed. 
Furthermore, even the participant who had the most experience in using large data sets (P10) had issues 
with cleaning the data to make it useable for their study. Second, teacher-focused EDS programs should 
address the different types of data analysis techniques available beyond descriptive statistics, including 
techniques of various levels of complexity, so that participants can shape their research questions to the 
required skills accordingly. Data analytics training should focus on techniques related to statistical 
analysis of the data and an overview of the tools available to visualize the results. 

6.3 Administrative and Participant Community Support 

6.3.1 Confirmation of Literature 

The participants noted that they valued two types of support received: support from the TEDS 
administration and support from the other program participants during the TEDS meetings. Although, 
participants predominantly received administrative support as opposed to support from the broader 
participant community. The support from the TEDS program was integral to the participant's experience, 
and although many encountered issues related to data literacy, the TEDS administrative support 
enabled the completion of the program's projects. These findings are similar to those that Ferguson et 
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al. (2014) noted in their Data Wranglers program, in which they determined that stakeholder 
participation throughout the program was critical. Furthermore, for the projects that made use of LMS 
data, the participants had to go outside of TEDS program support structure to access the data, which 
created issues of data access inequalities. As such, we believe that for the future development of TEDS 
and similar programs there should be an increased focus on developing the administrative support 
structure by including those with access to other forms of student data. This finding confirms what Clow 
(2014) found in the Data Wrangler program, where there were issues with the participants accessing 
and obtaining fine-grained and historical student data. Future teacher-focused EDS programs will 
require ongoing conversations between program participants, stakeholders, and data managers. 

6.3.2 New Insights 

We understand that the participants required support from the administrative community of TEDS for 
data access, but inter-project interactions may have provided avenues for the participants to 
collaboratively address data literacy issues, such as data analysis and data visualization techniques. It is 
our belief that collaboration between groups facilitated through the TEDS administration's fostering of a 
participant community would have further contributed to the projects and the participants' experience 
in the program. We do understand, however, that given the sensitivity of the data that many of the 
participants were using for their analyses and which required IRB approval, direct collaboration may not 
have been possible. However, a more firmly developed participant community would have enabled, as 
one of the participants put it, participants to "bounce ideas off of one another" (P6) and to assist one 
another in addressing the data literacy barriers that they faced. As Suthers and Verbert (2013) have 
argued, although learning analytics is multidisciplinary with various researchers having backgrounds in 
different theoretical and methodological traditions, it is important for the learning analytics community 
to embrace the "multivocality" of the domain to encourage dialogue amongst researchers. Even though 
Suthers and Verbert (2013) are specifically calling out the learning analytics research community, the 
same could be said for programs similar to TEDS in which a multidisciplinary group of educators come 
together as a part of a community to assist one another in the EDS. 

7 FUTURE DEVELOPMENT AND CONCLUSION 

We propose that future research on EDS programs that consider teachers as active participants in the 
data analytics process can benefit from the findings of this study. However, given that the work in 
teacher-focused EDS is nascent, future research is needed to better understand how teachers integrate 
the insights from teacher-generated analytics into learning design. This future EDS research should look 
to the contributions made in practitioner-based educational research. In practitioner research, the 
inquiry into the learning environment is conducted by teachers in order to evaluate their students and 
the learning environment and to improve their practice (Campbell & McNamara, 2010). In the 
practitioner research community of teacher inquiry into student learning (TISL), teachers use data 
generated from educational technologies to lead the research process in collaboration with educational 
researchers, technology designers, and institutional managers (Clark, Luckin, & Jewitt, 2011; Mor, 
Ferguson, & Wasson, 2015). There have been calls for integrating the research taking place in the 
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learning analytics community with TISL and other practitioner research communities, such as learning 
design (Mor et al., 2015), yet further research is needed on how to integrate these communities (Emin- 
Martinez et al., 2014). In addition, as teacher-focused EDS research continues to develop, researchers 
must begin to address how a teacher's ethical perspective on EDS influences their adoption and use of 
educational data analytics. As EDS scholars have noted, the ethical concerns of actors involved in EDS 
have the potential to hamper the implementation and use of educational data analytics, and that these 
concerns should be addressed during the design and implementation phases of EDS systems and 
programs (Pardo & Siemens, 2014; Slade & Prinsloo, 2013). 

In order to better understand EDS programs that focus on teachers as producers and contributors to the 
analytics process, we set out to identify the factors that both enable and constrain such programs in 
praxis. We have identified that university instructors and faculty are motivated to participate in EDS to 
answer questions related to learning analytics and academic analytics as opposed to being driven to 
engage in specific EDS communities, that the primary barriers teachers face are in regards to data 
literacy, and that both administrative and participant community support are important for enabling 
teacher-driven analytics projects. Focusing on these three areas can aid in the development of future 
EDS programs that are teacher-focused as opposed to being solely student-focused. We believe that this 
study adds to the small but growing body of research in the EDS community (Clow, 2014; Ferguson et al., 
2014), and can serve as a guide for educational practitioners considering implementing similar programs 
in the future, and for educational researchers interested further examining the teacher's role in student 
data analytics. 
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APPENDIX A 

Semi-Structured Interview Protocol 

1. How did you evaluate your classroom environment prior to TEDS? 

2. Had you ever used institutional student data prior to your involvement in the program? 

a. If yes, how have you used institutional data? 

3. What other evaluation resources, aside from institutional student data, have you used? 

4. Why was it important for you to be a part of TEDS? 

5. What did you hope to learn during your project? 

6. What did you learn from your project? 

7. Do you feel that the program will impact your classroom environment? 

a. If yes, how will the program impact your classroom environment? 

8. How do you plan to implement what you learned during the program? 

9. What barriers did you encounter during the program? 

10. Did you have the skills to conduct the analyses that you wanted? 

a. If no, what skills would you have better enabled your to complete your project? 

b. How did you address these limitations in your project? 

11. Were you able to get access to the data that you needed to conduct your analyses? 

a. If no, how was this accounted for in your project? 

12. In what ways could the program have addressed these issues? 

13. Is there anything you would like to add about your experience in the TEDS program? 
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